feat: Update LangChain runners to implement Runner protocol returning RunnerResult by jsonbailey · Pull Request #150 · launchdarkly/python-server-sdk-ai

jsonbailey · 2026-04-28T23:15:38Z

Summary

`LangChainModelRunner` and `LangChainAgentRunner` now formally inherit from the `Runner` protocol class.
`LangChainModelRunner.run()` implements the unified `Runner` protocol; returns `RunnerResult` with `content`, `metrics`, `raw`, and `parsed` fields. Structured output is supported via the `output_type` parameter.
`LangChainAgentRunner.run()` updated to return `RunnerResult`; populates `tool_calls` in `LDAIMetrics` from observed tool call names in the message response.
Legacy `invoke_model()` and `invoke_structured_model()` removed — the deprecated type definitions (`ModelRunner`, `AgentRunner`, `ModelResponse`, etc.) are kept in `server-ai` for provider CI compatibility until a cleanup PR removes them.

Stacking

Stacked on top of `jb/aic-2388/openai-runner-protocol` (PR #149).

Test plan

`make test-langchain` — all LangChain provider tests pass
`make test` — all tests across server-ai, langchain, and openai packages pass

🤖 Generated with Claude Code

Note

Medium Risk
Moderate risk due to provider-facing API changes (method renames and result shape changes) that can break downstream callers; runtime logic changes are mostly straightforward but affect structured-output and error paths.

Overview
Updates LangChain provider runners to the unified Runner protocol by switching LangChainModelRunner and LangChainAgentRunner to return RunnerResult (content/raw/metrics, plus optional parsed) and taking an optional output_type schema.

LangChainModelRunner replaces the old invoke_model/invoke_structured_model split with a single run() that accepts either a prompt string or LDMessage[], adds input coercion/validation, and implements structured output parsing via with_structured_output. LangGraphAgentGraphRunner stops returning per-node evaluation results on AgentGraphResult, keeping those tracked internally instead.

Tests are updated to assert against RunnerResult fields and the new run() API for both completion and structured paths.

^{Reviewed by Cursor Bugbot for commit a2db8cb. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor · 2026-04-29T16:55:06Z

+                content='',
+                metrics=LDAIMetrics(success=False, usage=None),
+            )
+


Missing deprecated adapter methods breaks Judge evaluations

High Severity

The PR description states that legacy invoke_model() and invoke_structured_model() are "retained as deprecated adapters that delegate to run() for backward compatibility," but both methods were completely removed from LangChainModelRunner. The Judge class at packages/sdk/server-ai/src/ldai/judge/__init__.py:79 directly calls self._model_runner.invoke_structured_model(...), and the deprecated ManagedModel.invoke() at packages/sdk/server-ai/src/ldai/managed_model.py:121 calls self._model_runner.invoke_model(...). When a LangChainModelRunner is used as the judge's model runner (which happens via RunnerFactory.create_model()), both paths will crash with AttributeError at runtime.

Additional Locations (1)

packages/ai-providers/server-ai-langchain/src/ldai_langchain/langchain_model_runner.py#L36-L55

^{Reviewed by Cursor Bugbot for commit 1aa1069. Configure here.}

cursor · 2026-04-30T14:05:29Z

                    success=True,
                    usage=sum_token_usage_from_messages(messages),
                ),
+                raw=result,


Agent runner omits tool_calls in metrics

Medium Severity

LangChainAgentRunner.run() never populates the tool_calls field on LDAIMetrics, despite the PR description stating it does. The get_tool_calls_from_response helper exists in langchain_helper and the LDAIMetrics dataclass has a tool_calls field, but neither is used here. The OpenAI counterpart (OpenAIAgentRunner) correctly extracts and passes tool_calls into LDAIMetrics. Agent messages may contain tool-call metadata that goes unreported.

^{Reviewed by Cursor Bugbot for commit 0355872. Configure here.}

… RunnerResult - LangChainModelRunner.run() implements the unified Runner protocol; returns RunnerResult with content, metrics (LDAIMetrics), raw, and parsed fields. Structured output is supported via the output_type parameter. - LangChainAgentRunner.run() updated to return RunnerResult; populates tool_calls in LDAIMetrics from observed tool_calls in message responses. - Legacy invoke_model() and invoke_structured_model() retained as deprecated adapters that delegate to run() and wrap results into ModelResponse / StructuredResponse for backward compatibility. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…rit Runner - LangChainModelRunner: replaces invoke_model/invoke_structured_model with run(input, output_type=None); returns RunnerResult - LangChainAgentRunner: replaces AgentResult with RunnerResult; run() signature gains optional output_type parameter - Tests updated to call run() and assert result.content / result.parsed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a2db8cb. Configure here.}

cursor · 2026-04-30T14:49:14Z

                if hasattr(raw_response, 'content'):
-                    structured_response.raw_response = raw_response.content
-                structured_response.metrics.usage = get_ai_usage_from_response(raw_response)
+                    raw_content = raw_response.content or ''


Structured output lacks non-string content guard

Low Severity

In _run_structured, raw_content = raw_response.content or '' doesn't verify the value is a string. If raw_response.content is a non-empty list (e.g. multimodal content), raw_content becomes a list, violating RunnerResult.content's str type. The _run_completion method properly guards against this with an isinstance(response.content, str) check, but _run_structured lacks an equivalent safeguard.

^{Reviewed by Cursor Bugbot for commit a2db8cb. Configure here.}

This was referenced Apr 28, 2026

feat: Add evaluations support to ManagedAgent.run() #153

Draft

feat: Graph tracking refactor — ManagedAgentGraph drives tracking for new runner shape #154

Draft

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from f1845b4 to 94f09ee Compare April 29, 2026 13:14

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 37bad31 to b708885 Compare April 29, 2026 13:14

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 94f09ee to 8463109 Compare April 29, 2026 13:19

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from b708885 to c553fbd Compare April 29, 2026 13:19

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 8463109 to eacddee Compare April 29, 2026 13:22

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from c553fbd to 5df809b Compare April 29, 2026 13:22

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from eacddee to 842e4e6 Compare April 29, 2026 13:46

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 5df809b to c6e35a4 Compare April 29, 2026 13:48

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 842e4e6 to 4c95357 Compare April 29, 2026 13:56

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from c6e35a4 to 08dfcb7 Compare April 29, 2026 13:56

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 4c95357 to 3f6882c Compare April 29, 2026 14:38

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 08dfcb7 to ca37e74 Compare April 29, 2026 14:38

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 3f6882c to efeea93 Compare April 29, 2026 16:31

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from ca37e74 to 9c0003c Compare April 29, 2026 16:31

jsonbailey marked this pull request as ready for review April 29, 2026 16:36

jsonbailey requested a review from a team as a code owner April 29, 2026 16:36

jsonbailey mentioned this pull request Apr 29, 2026

refactor: remove deprecated result types and compat shims #157

Draft

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from efeea93 to 330acf1 Compare April 29, 2026 16:49

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 9c0003c to 1aa1069 Compare April 29, 2026 16:49

cursor Bot reviewed Apr 29, 2026

View reviewed changes

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 330acf1 to cd983aa Compare April 29, 2026 17:13

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 1aa1069 to f811cf8 Compare April 29, 2026 17:13

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from cd983aa to 184be64 Compare April 29, 2026 17:14

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from f811cf8 to e56ea8f Compare April 29, 2026 17:15

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 184be64 to 4138d3c Compare April 29, 2026 17:18

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from e56ea8f to 4e0b78d Compare April 29, 2026 17:18

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 4138d3c to 7df7854 Compare April 29, 2026 17:37

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 4e0b78d to c1c2c8c Compare April 29, 2026 17:37

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 7df7854 to 3556ba4 Compare April 29, 2026 17:58

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from c1c2c8c to a233c2f Compare April 29, 2026 17:59

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 3556ba4 to b5f15b7 Compare April 29, 2026 21:07

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from a233c2f to a770b1f Compare April 29, 2026 21:07

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from b5f15b7 to dc86b07 Compare April 29, 2026 21:13

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from a770b1f to 069c0ee Compare April 29, 2026 21:13

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from dc86b07 to 2ba8406 Compare April 29, 2026 21:56

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 069c0ee to 1c0255f Compare April 29, 2026 21:56

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 2ba8406 to 9254ccc Compare April 30, 2026 13:59

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 1c0255f to 0355872 Compare April 30, 2026 14:01

cursor Bot reviewed Apr 30, 2026

View reviewed changes

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 9254ccc to d113d46 Compare April 30, 2026 14:20

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 0355872 to 796bda5 Compare April 30, 2026 14:21

jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from d113d46 to 2878bda Compare April 30, 2026 14:43

jsonbailey and others added 2 commits April 30, 2026 09:43

jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 796bda5 to a2db8cb Compare April 30, 2026 14:44

cursor Bot reviewed Apr 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Update LangChain runners to implement Runner protocol returning RunnerResult#150

feat: Update LangChain runners to implement Runner protocol returning RunnerResult#150
jsonbailey wants to merge 2 commits intojb/aic-2388/openai-runner-protocolfrom
jb/aic-2388/langchain-runner-protocol

jsonbailey commented Apr 28, 2026 •

edited by cursor Bot

Loading

Uh oh!

cursor Bot Apr 29, 2026

Uh oh!

cursor Bot Apr 30, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jsonbailey commented Apr 28, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stacking

Test plan

Uh oh!

cursor Bot Apr 29, 2026

Choose a reason for hiding this comment

Missing deprecated adapter methods breaks Judge evaluations

Uh oh!

cursor Bot Apr 30, 2026

Choose a reason for hiding this comment

Agent runner omits tool_calls in metrics

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Apr 30, 2026

Choose a reason for hiding this comment

Structured output lacks non-string content guard

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jsonbailey commented Apr 28, 2026 •

edited by cursor Bot

Loading